CHAPTER 3 Getting Statistical: A Short Review of Basic Statistics 43

sampling. You may never know what that truth is, but an objective truth is out

there nonetheless.

The truth can be one of two answers — the effect is there, or the effect is not

there. Also, your conclusion from your sample will be one of two answers — the

effect is there, or the effect is not there.

These factors can be combined into the following four situations:»

» ^{Your test is not statistically significant, and H}0 ^{is true. This is an ideal}

situation because the conclusion of your test matches the truth. If you were

testing the mean difference in effect between Drug A and Drug B, and in truth

there was no difference in effect, if your test also was not statistically signifi-

cant, this would be an ideal result.»

» ^{Your test is not statistically significant, but H}0 ^{is false. In this situation, the}

interpretation of your test is wrong and does not match truth. Imagine testing

the difference in effect between Drug C and Drug D, where in truth, Drug C

had more effect than Drug D. If your test was not statistically significant, it

would be the wrong result. This situation is called Type II error. The probability

of making a Type II error is represented by the Greek letter beta (β).»

» ^{Your test is statistically significant, and H}Alt ^{is true. This is another}

situation where you have an ideal result. Imagine we are testing the difference

in effect between Drug C and Drug D, where in truth, Drug C has more of an

effect. If the test was statistically significant, the interpretation would be to

reject H0, which would be correct.»

» ^{Your test is statistically significant, but H}Alt ^{is false. This is another}

situation where your test interpretation does not match the truth. If there was

in truth no difference in effect between Drug A and Drug B, but your test was

statistically significant, it would be incorrect. This situation is called Type I

error. The probability of making a Type I error is represented by the Greek

letter alpha (α).

We discussed setting α = 0.05, meaning that you are willing to tolerate a Type I

error rate of 5 percent. Theoretically, you could change this number. You can

increase your chance of making a Type I error by increasing your α from 0.05 to a

higher number like 0.10, which is done in rare situations. But if you reduce your α

to number smaller than 0.05 — like 0.01, or 0.001 — then you run the risk of never

calculating a test statistic with a p value that is statistically significant, even if a

true effect is present. If α is set too low, it means you are being very picky about

accepting a true effect suggested by the statistics in your sample. If a drug really

is effective, you want to get a result that you interpret as statistically significant

when you test it. What this shows is that you need to strike a balance between the